Remote Commands

URL has a set of default commands that are used to issue actions on the remote. While these work for most situations, they can be changed in the event that they don’t.

Python, Submitter and Shell

URL has 3 main running commands:

  1. python: Used to launch a python script, defaults to python

  2. submitter: Used to submit the jobscript, defaults to bash (direct submission)

  3. shell: Used to change the command which is used to run the master script, defaults to bash

Of these, python and submitter are the most important.

Shell exists for an edge case where you need a specific command to run scripts, and can usually be left at its default.

Note

If submitter is left default, yet shell has been changed, url.submitter will return the value of shell. For example, setting url.shell = "sh" with a default submitter will cause url.submitter to return "sh"

Lets set up a quick example that shows what changes are made when all three of these are set:

[1]:

from remotemanager import Dataset, URL

url = URL(python='python3', submitter='sbatch', shell = 'my_shell')

def f():
    return

ds = Dataset(f, url=url, skip=False)

ds.append_run()

appended run runner-0

The run command is where the shell takes effect, it is what is used to execute the master script.

[2]:

print("run command:")
ds.run(dry_run=True)

run command:
Running Dataset
assessing run for runner dataset-06a84b6d-runner-0... running
launch command: cd temp_runner_remote && my_shell dataset-06a84b6d-master.sh

The contents of the master script are somewhat confusing to read.

Since the master script deals with the actual job submission, there is extra content in there to make sure the jobs run properly.

The important thing for us here is the sbatch dataset_... section in the middle of the long line.

[3]:

print('master script:')
print(ds.master_script.content)

master script:
sourcedir=$PWD
rm -f dataset-06a84b6d.manifest
echo $(date +'%d/%m/%Y %H:%M:%S') 66c0efed submitted >> dataset-06a84b6d.manifest && sed -i -e "s#66c0efed_master#$sourcedir#" dataset-06a84b6d-runner-0-jobscript.sh &&  sbatch dataset-06a84b6d-runner-0-jobscript.sh 2> dataset-06a84b6d-runner-0-error.out

The jobscript is what deals with the actual job execution. Here it is nice and simple, the final line is using our python3 specification.

[4]:

print('jobscript:')
print(ds.runners[0].jobscript.content)

jobscript:
#!/bin/bash


export DIR_66c0efed=66c0efed_master
python3 dataset-06a84b6d-runner-0-run.py 2>> dataset-06a84b6d-runner-0-error.out